Toward the Scalable Integration of Internet

This dissertation in a broad sense focuses on understanding the fundamental aspects of building a large-scale information integration system that can answer complex queries over a large number of heterogeneous Internet data sources. Among many challenges in achieving this goal, we focus on two key issues: efficient query processing and schema matching. Most of the data the integration system processes arrives in a stream from a remote source rather than residing on a local disk; we need to develop efficient query processing algorithms that work in this environment. We specifically investigate algorithms for evaluating sliding window joins over pairs of unbounded streams. We introduce a unit-time-basis cost model to analyze the expected performance of these algorithms. Using this cost model, we propose strategies for maximizing the efficiency of processing joins over unbounded streams. With respect to schema matching, we introduce an automated technique that works in the particularly difficult cases in which attribute names and data values are “opaque.” While schema matching has always been a problematic and interesting aspect of information integration, the problem is exacerbated as the number of information sources to be integrated grows. Our approach does not depend on any domain specific knowledge, and hence is applicable to many different domains, even including domains to which the system has not previously been exposed. Our approach works in two steps: 1) we measure the pair-wise attribute correlations in the tables to be matched and construct a dependency graph using mutual information as a measure of the ii dependency between attributes, and then 2) we find matching node pairs in the dependency graphs by running a graph matching algorithm. We validate our approach with an experimental study.